For any suggestions or feedback regarding these notes,

please contact Pragy Agarwal

Video streaming

Challenges with video streaming

Video content is very large & bandwidth heavy

a video can range from a few 100 MBs to 10s of GBs

Multiple clients can stream the same video at the same time

For a popular Netflix show like Game of Thrones, there might be hundreds of thousands of people streaming the same video simultaneously
For live content (like Cricket matches), the concurrency numbers can be crazy – 60M+ concurrent viewers for the India-Pak match

Different clients have different hardware & software configurations

different Screen resolutions

1080p @ 60hz, 2k @ 144 hz, 4k @ 144hz, …

different CPU/GPU configurations

ARM chips, desktop CPUs, integrated graphics, gaming GPUs, custom TV chipsets, …

Different clients have different network bandwidths

1Mbps to 1Gbps

Internet speed of the same client can vary over time

Especially for mobile users

connect to a different tower
entering a Faraday Shield — inside elevator, under a flyover, … can cause your internet speeds to suffer

But also for smart-TV / desktop users, because they might share a network connection with others

Adaptive Bitrate Streaming (ABS)

1. Should all clients stream at the same resolution – say 1080p?

No!

Different clients have different internet speeds.

If we force all clients to

Stream at highest resolution — clients with slower bandwidth will suffer, since their video will “buffer”

Stream at lowest resolution — clients with higher bandwidths won’t be able to enjoy their cat videos

Solution: we should serve different resolutions to different clients based on their bandwidths!

pre-process (in the backend) the uploaded video to various resolutions & fps

input

one_piece_1080p_60fps.mp4

output

one_piece_720p_60fps.mp4 one_piece_720p_30fps.mp4

one_piece_480p_60fps.mp4 one_piece_480p_30fps.mp4

one_piece_360p_60fps.mp4 one_piece_360p_30fps.mp4

one_piece_240p_60fps.mp4 one_piece_240p_30fps.mp4

This is called downsampling.

Could you also upsample the video? Nope. (maybe with AI – it will still be a lie)

serve the video at the appropriate resolution

Q: Won’t this increase the amount of storage required in the backend by several times?

Storage is cheap - you shouldn’t worry about it.

At the scale of youtube, it starts mattering

Youtube stores more than 1 exabytes!

Actually netflix’s content repository is much smaller

Lower resolutions also require drastically lower storage

2 hour video at 8k @ 60fps might be 15GB in size (compressed)
2 hour video at 1080p @ 30fps might be only 1.5 GB in size (compressed)
Overall if you add the file sizes for all the downsampled resolution, your total storage requirement will grow to only ~2x

1 + ½ + ¼ + ⅛ + … = 2

2.1. How do we determine the best resolution for the client?

In the frontend, we can monitor the client’s bandwidth, and serve the highest resolution that will not buffer on their bandwidth.

Maintain a mapping of bandwidth to resolution

Resolutions	Required Bandwidth (H264)
720p – 1280×720 (HD)	3Mbps
1080p – 1920X1080 (FHD)	6Mbps
2160p / 4K – 3840×2160 (UHD)	25Mbps

Note: actual requirements will depend on the video codec. For example, H265 requires almost ½ the bandwidth compared to H264

Note: you typically don’t want to enforce a resolution on the client — you just detect and configure the “default” resolution, but the client should be able to override this via preferences

2.2. Given a client that has 1Gbps network & 720p screen, should we serve them a video at 1080p?

The client’s bandwidth can handle 1080p easily — so yes?

No! Screen supports max 720p resolution, so no point in serving a higher resolution to the client, it will just waste their data pack.

In fact, the client’s CPU/GPU will have to work harder to downsample the 1080p video to a 720p screen.

The actual resolution that the client sees should depend not only on the bandwidth, but also on the hardware configuration

CPU/GPU: video decoding requires intensive processing. High resolution videos can “stutter” on low end devices
Screen Size & Refresh rate: we should not serve at a resolution or fps that is higher than what the client’s screen supports
Software configuration: depending on the browser & video codecs installed, we might want to serve video in different formats

2.3. How do we get the client’s bandwidth and hardware & software configuration?

Bandwidth: perform a speed test

download a junk (random) file of known size
measure the time it took for the download to complete
bandwidth = file size / time

eg, 100 MB file takes 10 seconds to download, then the bandwidth is 10MBps

Hardware & Software Configuration

User Agent detection

console.log(navigator.userAgent)

'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36'

Device Fingerprinting
cookies, operating system, http headers, laptop battery %, user behavior (clicks, mouse movements, keyboard strokes), … lots of other stuff

3.1. Should we download the entire video before playing?

Long videos at high resolutions can have very large file size (10s of GBs). If we download the entire video before playback, the client will have to wait for a long time.

Instead, we should “stream” the video — play the video as it is being downloaded.

3.2. What if the client’s “seeks” forward/backward during the video?

Suppose a client starts watching a video, but skips to the 30 min mark. How do we handle that?

Re-download the entire video from scratch?
That will be a waste of bandwidth, and will require a lot of time
Download video from that timestamp
But the video is a binary file — how do we know the bit offset from the timestamp?

A 2 hour movie @ 8k resolution @ 60fps, will cost you ~43 TB in the raw format!

Solution: chunk the video (at each resolution) into small sections.

input

one_piece_1080p_60fps.mp4

output

one_piece_720p_60fps_chunk_01.mp4 one_piece_720p_30fps_chunk_01.mp4

one_piece_720p_60fps_chunk_02.mp4 one_piece_720p_30fps_chunk_02.mp4

one_piece_720p_60fps... one_piece_720p_30fps...

one_piece_720p_60fps_chunk_20.mp4 one_piece_720p_30fps_chunk_20.mp4

one_piece_480p_60fps_chunk_01.mp4 one_piece_480p_30fps_chunk_01.mp4

one_piece_480p_60fps_chunk_02.mp4 one_piece_480p_30fps_chunk_02.mp4

one_piece_480p_60fps... one_piece_480p_30fps...

one_piece_480p_60fps_chunk_20.mp4 one_piece_480p_30fps_chunk_20.mp4

one_piece_360p_60fps_chunk_01.mp4 one_piece_360p_30fps_chunk_01.mp4

one_piece_360p_60fps_chunk_02.mp4 one_piece_360p_30fps_chunk_02.mp4

one_piece_360p_60fps... one_piece_360p_30fps.,..

one_piece_360p_60fps_chunk_20.mp4 one_piece_360p_30fps_chunk_20.mp4

Client tries to watch the video
Video player on the frontend will determine the “best” resolution to serve the video at
Video player (client side) will download the first chunk at the appropriate resolution
Once the first chunk is downloaded, and the video playback will start — the video player will start playing this chunk
While the video playback continues, the video player (client side) will keep downloading the next chunks.
If the client seeks forward to a specific timestamp, we can directly request the last chunk starting before or at that timestamp
Any chunk that is downloaded will remain in the browser cache — allows the client to seek to an older timestamp without re-buffering

3.3. What is the ideal chunk size?

Should we chunk by number of bits? 10MB / 100MB … ?

Should we chunk by the time duration? 10 sec / 1 min

Implications of chunk size

higher chunk size will cause more initial buffering, since the client has to wait longer to download the 1st chunk
lower chunk size will cause more n/w requests, since each chunk has to be downloaded separately

Ideal chunk size will depend on the application (more on this later)

4.1. What if the client’s bandwidth changes during streaming?

We should continuously monitor the client’s bandwidth.

But how? Should we regularly download junk files to measure the bandwidth?

If we do this, won’t we end up wasting precious bandwidth and data?

As the chunks are being downloaded, we can measure the time taken for download. Since each chunk has a known byte size (stored in the file metadata), we can calculate the bandwidth continuously!

Note: each chunk has the same duration (say 10s), but different file sizes (based on how complex the video scene is)

Each next chunk will be downloaded at the appropriate resolution, given the latest bandwidth.

Note that all this calculation & bandwidth monitoring is happening in the client side.

async getChunk(url: string): Promise<{data: ByteArray[], size: integer, timeMs: integer}> {

startTime = new Date()

response = await get(url => …)

timeMs = new Date() - startTime()

}

4.2 Should we download the next chunk immediately after the previous one is downloaded?

No. Most people don’t watch the entire video – they skip ahead, or switch to another video.

We should download the next chunk “Just in Time” — download the next chunk when

remaining-playback-time <= (time-to-download-next-chunk + x% buffer )

By the time the user reaches the end of available playback, the next chunk should be available — so that we don’t waste any bandwidth, and the client doesn’t need to buffer

HTTP(s) Live Streaming (HLS)

HTTP(s) based Adaptive Bitrate Streaming protocol, introduced by Apple in 2009
Uses XML based .m3u8 files
Most popular video streaming protocol (as of 2026)
Supports encryption & copyright protection
Supports ads, section markers (intro, credits, …), thumbnails, …

.m3u8 Demo

Base .m3u8

https://demo.unified-streaming.com/k8s/features/stable/video/tears-of-steel/tears-of-steel.ism/.m3u8

#EXTM3U

#EXT-X-VERSION:1

## Created with Unified Streaming Platform (version=1.13.0-29687)

# variants

#EXT-X-STREAM-INF:BANDWIDTH=493000,CODECS="mp4a.40.2,avc1.66.30",RESOLUTION=224x100,FRAME-RATE=24

tears-of-steel-audio_eng=64008-video_eng=401000.m3u8

#EXT-X-STREAM-INF:BANDWIDTH=932000,CODECS="mp4a.40.2,avc1.66.30",RESOLUTION=448x200,FRAME-RATE=24

tears-of-steel-audio_eng=128002-video_eng=751000.m3u8

#EXT-X-STREAM-INF:BANDWIDTH=1197000,CODECS="mp4a.40.2,avc1.77.31",RESOLUTION=784x350,FRAME-RATE=24

tears-of-steel-audio_eng=128002-video_eng=1001000.m3u8

#EXT-X-STREAM-INF:BANDWIDTH=1727000,CODECS="mp4a.40.2,avc1.100.40",RESOLUTION=1680x750,FRAME-RATE=24,VIDEO-RANGE=SDR

tears-of-steel-audio_eng=128002-video_eng=1501000.m3u8

#EXT-X-STREAM-INF:BANDWIDTH=2468000,CODECS="mp4a.40.2,avc1.100.40",RESOLUTION=1680x750,FRAME-RATE=24,VIDEO-RANGE=SDR

tears-of-steel-audio_eng=128002-video_eng=2200000.m3u8

# variants

#EXT-X-STREAM-INF:BANDWIDTH=68000,CODECS="mp4a.40.2"

tears-of-steel-audio_eng=64008.m3u8

#EXT-X-STREAM-INF:BANDWIDTH=136000,CODECS="mp4a.40.2"

tears-of-steel-audio_eng=128002.m3u8

Specific Resolution — 1680x750 @ 24FPS

https://demo.unified-streaming.com/k8s/features/stable/video/tears-of-steel/tears-of-steel.ism/tears-of-steel-audio_eng=128002-video_eng=2200000.m3u8

#EXTM3U

#EXT-X-VERSION:1

## Created with Unified Streaming Platform (version=1.13.0-29687)

#EXT-X-MEDIA-SEQUENCE:1

#EXT-X-TARGETDURATION:4

#USP-X-TIMESTAMP-MAP:MPEGTS=900000,LOCAL=1970-01-01T00:00:00Z

#EXTINF:4, no desc

tears-of-steel-audio_eng=128002-video_eng=2200000-1.ts

#EXTINF:4, no desc

tears-of-steel-audio_eng=128002-video_eng=2200000-2.ts

#EXTINF:4, no desc

tears-of-steel-audio_eng=128002-video_eng=2200000-3.ts

#EXTINF:4, no desc

tears-of-steel-audio_eng=128002-video_eng=2200000-4.ts

...

#EXTINF:4, no desc

tears-of-steel-audio_eng=128002-video_eng=2200000-183.ts

#EXTINF:2, no desc

tears-of-steel-audio_eng=128002-video_eng=2200000-184.ts

#EXT-X-ENDLIST

Specific Chunk — 1st

https://demo.unified-streaming.com/k8s/features/stable/video/tears-of-steel/tears-of-steel.ism/tears-of-steel-audio_eng=128002-video_eng=2200000-1.ts

(binary file – can be played in your video player)

Resources on HLS (optional)

Streaming Pre-recorded videos

Youtube / Netflix / JioCinema

Step 1. Content creator uploads the video

The video is uploaded at the upload microservice, which stores in the database.

Ideal database?

video: Large File Store (s3)
metadata: SQL (or Doc DB is there’s too much data)

schema: for each video id store chunk ids, resolutions, corresponding chunk URLs, uploader information, any other metadata about the video
joins: given the uploader_id, find all videos by them
complex queries: find all 720p videos uploaded in the last 1 hour

Step 2: Video is processed into multiple resolutions — downsampling & chunking

Apart from inserting the video into S3 & metadata into SQL, the upload service will also insert a task in a Message Queue.

Downsampling service picks tasks from the message queue. For each task,

Fetch Video from FileStore, fetch Metadata from SQL
Downsample the video to various resolutions
Chunk each video in multiple segments
Upload the chunks to file store
Create & store metadata for all these chunks in SQL

input

one_piece_1080p_60fps.mp4

output

one_piece_720p_60fps_chunk_01.mp4 one_piece_720p_30fps_chunk_01.mp4

one_piece_720p_60fps... one_piece_720p_30fps...

one_piece_720p_60fps_chunk_20.mp4 one_piece_720p_30fps_chunk_20.mp4

one_piece_480p_60fps_chunk_01.mp4 one_piece_480p_30fps_chunk_01.mp4

one_piece_480p_60fps... one_piece_480p_30fps...

one_piece_480p_60fps_chunk_20.mp4 one_piece_480p_30fps_chunk_20.mp4

one_piece_360p_60fps_chunk_01.mp4 one_piece_360p_30fps_chunk_01.mp4

one_piece_360p_60fps... one_piece_360p_30fps...

one_piece_360p_60fps_chunk_20.mp4 one_piece_360p_30fps_chunk_20.mp4

Step 3: Serving the video to clients

We need to serve both the video chunks (.ts files) & the metadata (.m3u8 file)

Should the clients download the video chunks from our file storage service?

No! File Storage Services like S3 are not optimized for fast reads/writes at high scale.

Instead the user will download the chunks from the CDN (always).

“ CDNs perform the heavy lifting for serving videos @ scale ”

What about the metadata?

If the metadata (m3u8) file is static, it can also be served from the CDN.

However, it can be dynamic as well

dynamic advertisement injection
geolocation based content personalization

In such cases, we have to “construct” the m3u8 for each request via a backend app server.

The app server can depend on a caching layer to store the constructed m3u8, instead of fetching the metadata from the SQL db & reconstructing the file every time.

Ideal Chunk size

For pre-recorded content, the chunk sizes can be larger — typically kept between 5s to 10s to reduce the number of network calls.

“ Apple recommends 6s chunk size for HLS ”

In some services, the chunk size can be as large as 1 minute

Scaler Recorded Video (not live class) have a chunk size of 16.5s

Video streaming

Challenges with video streaming

Adaptive Bitrate Streaming (ABS)

1. Should all clients stream at the same resolution – say 1080p?

Q: Won’t this increase the amount of storage required in the backend by several times?

2.1. How do we determine the best resolution for the client?

2.2. Given a client that has 1Gbps network & 720p screen, should we serve them a video at 1080p?

2.3. How do we get the client’s bandwidth and hardware & software configuration?

3.1. Should we download the entire video before playing?

3.2. What if the client’s “seeks” forward/backward during the video?

3.3. What is the ideal chunk size?

4.1. What if the client’s bandwidth changes during streaming?

4.2 Should we download the next chunk immediately after the previous one is downloaded?

HTTP(s) Live Streaming (HLS)

.m3u8 Demo

Base .m3u8

Specific Resolution — 1680x750 @ 24FPS

Specific Chunk — 1st

Resources on HLS (optional)

Popular Video Platforms

Streaming Pre-recorded videos

Step 1. Content creator uploads the video

Ideal database?

Step 2: Video is processed into multiple resolutions — downsampling & chunking

Step 3: Serving the video to clients

Should the clients download the video chunks from our file storage service?

What about the metadata?

Ideal Chunk size